With the rapid development of edge intelligence, the development of existing handwritten numeral recognition convolutional network models has become less and less suitable for the requirements of edge deployment and computing power declining, and there are problems such as poor generalization ability of small samples and high network training costs. Drawing on the classic structure of Convolutional Neural Network (CNN), Leaky_ReLU algorithm, dropout algorithm, genetic algorithm and adaptive and mixed pooling ideas, a handwritten numeral recognition model based on LeNet-DL improved convolutional neural network was constructed. The proposed model was compared on large sample MNIST dataset and small sample REAL dataset with LeNet, LeNet+sigmoid, AlexNet and other algorithms. The improved network has the large sample identification accuracy up to 99.34%, with the performance improvement of about 0.83%, and the small sample recognition accuracy up to 78.89%, with the performance improvement of about 8.34%. The experimental results show that compared with traditional CNN, LeNet-DL network has lower training cost, better performance and stronger model generalization ability on large sample and small sample datasets.